Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk
نویسندگان
چکیده
In this paper we present results from two pilot studies which show that using the Amazon Mechanical Turk for preposition error annotation is as effective as using trained raters, but at a fraction of the time and cost. Based on these results, we propose a new evaluation method which makes it feasible to compare two error detection systems tested on different learner data sets.
منابع مشابه
Error Driven Paraphrase Annotation using Mechanical Turk
The source text provided to a machine translation system is typically only one of many ways the input sentence could have been expressed, and alternative forms of expression can often produce a better translation. We introduce here error driven paraphrasing of source sentences: instead of paraphrasing a source sentence exhaustively, we obtain paraphrases for only the parts that are predicted to...
متن کاملCan Crowdsourcing be used for Effective Annotation of Arabic?
Crowdsourcing has been used recently as an alternative to traditional costly annotation by many natural language processing groups. In this paper, we explore the use of Amazon Mechanical Turk (AMT) in order to assess the feasibility of using AMT workers (also known as Turkers) to perform linguistic annotation of Arabic. We used a gold standard data set taken from the Quran corpus project annota...
متن کاملThe Wisdom of the Crowdâs Ear: Speech Accent Rating and Annotation with Amazon Mechanical Turk
Human listeners can almost instantaneously judge whether or not another speaker is part of their speech community. The basis of this judgment is the speaker’s accent. Even though humans judge speech accents with ease, it has been tremendously difficult to automatically evaluate and rate accents in any consistent manner. This paper describes an experiment using the Amazon Mechanical Turk to deve...
متن کاملLearning From Noisy Singly-labeled Data
Supervised learning depends on annotated examples, which are taken to be the ground truth. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant ...
متن کاملEfficiently Scaling Up Video Annotation with Crowdsourced Marketplaces
[1] Yuen, J., Russell, B., Liu, C., Torralba, A.: LabelMe video: Building a Video Database with Human Annotations. (2009) [2] Vijayanarasimhan, S., Grauman, K.: Whats It Going to Cost You?: Predicting Effort vs. Informativeness for Multi-Label Image Annotations, CVPR (2009) [3] Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. Urbana 51 (2008) 61820 Interactive Vide...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2010